Enhanced Algorithm for the Detection of Eye Motion from Fundus Images

ABSTRACT

A method, controller, and non-transitory medium for obtaining images of a moving subject. Estimating changes in position of the subject based on the images. Calculating a quality metric of the estimation of the change. Comparing the quality metric to a threshold. In a first case in which the quality metric is less than the threshold, adjusting position of the scanning area based on the estimated change in position. In a second case in which the quality metric is not less than the threshold, obtaining a new second image.

GOVERNMENT LICENSE RIGHTS

This invention was made with government support under grant numbersEY014375 and EY001319 awarded by the National Institutes of Health. TheGovernment has certain rights in the invention.

BACKGROUND

Field of Art

The present disclosure relates to a system and method for controlling anopthalmoscope.

Description of the Related Art

Ophthalmoscopes, ophthalmic image pickup apparatuses, fundus imagingsystems such as: scanning laser ophthalmoscopes (SLOs) that irradiatethe fundus with a laser in two dimensions; and optical coherencetomographs (OCTs) that utilizes the interference of low coherence lighthave been developed and commercialized. Thus, SLOs and OCTs have becomeimportant tools for the study of the human fundus in both normal anddiseased eyes. Eye movement is a big issue for these imaging devices.

For example, a SLO may take multiple images for averaging andconstructing panoramic images. For constructing these panoramic images,each image should be at a precise position. This can be difficult toaccomplish because the eye moves continuously during imaging.Especially, on small FOV (Field of View) system such as AO-SLO (AdaptiveOptics SLO) eye movement is quite large when compared with the imagesize of the images and sometimes the imaging area can go out of frameeasily due to eye movement.

The robustness of the typical eye tracking process may run intodifficulties due to incorrect detection of eye motion when dealing withlow contrast images from a diseased eye. Images of a diseased eye areoften accompanied by high noise due to low reflection from the diseasedareas. This issue is exacerbated by uncertainty of the peak locationfrom cross correlation calculations when random noise negativelydisturbs image features, and/or image features and image background donot provide sufficient information for the cross correlation.Unsuccessful tracking becomes worse particularly when the referenceimage and the target image (image to be correlated) have long temporalinterval where parameters such as amount of torsion,brightness/contrast, and noise level of retinal images keeps changing.

What is needed is an eye position tracking system that is both fast androbust. The applicants have developed a process that addresses thisproblem. The process detects the eye position in a robust manner with aposition detection system and shifts imaging area according to the eyemovement with tracking mirrors.

SUMMARY

An aspect of an embodiment is a method of imaging a scanning area of asubject by scanning imaging light over periods of time on the scanningarea of the subject. The method may include constructing a plurality ofimages based on detection of the imaging light from the scanning area ofthe subject, each image among the plurality of images obtained during adifferent time period. The plurality of images may include at least: areference image, a first image, and a second image. The first image maybe obtained during a first period of time, the second image may beobtained during a second period of time, the reference image may beobtained during a third period of time that is prior to the first periodof time and the second period of time. The method may include estimatinga first relative change in position of the scanning area of the subjectby comparing the reference image and the first image. The method mayinclude estimating a second relative change in position of the scanningarea of the subject by comparing the reference image and the secondimage. The method may include estimating a third relative change inposition of the scanning area of the subject by comparing the firstimage and the second image. The method may include calculating a qualitymetric of estimation of the second relative change in position based on:the first relative change in position; the second relative change inposition; the third relative change in position. The method may includecomparing the quality metric to a threshold. In a first case in whichthe quality metric is less than the threshold, the method may includeadjusting position of the scanning area based on the second relativechange in position. In a second case in which the quality metric is notless than the threshold, the method may include obtaining a new secondimage during a new second period of time.

In an alternative embodiment the method may include, in the second case,the position of the scanning area may be controlled based upon previousimage position information.

In an alternative embodiment the method may include, the second imagemay be an image of the scanning area of the subject that is obtainedimmediately after the first image is obtained.

In an alternative embodiment, in the first case the method may includesetting the second image as a new first image; setting the secondrelative change in position as a new first relative change in position;obtaining the new second image during the new second period of time,wherein the new second image is an image of the scanning area of thesubject that is obtained immediately after the new first image isobtained; estimating a new second relative change in position of thescanning area of the subject by comparing the reference image and thenew second image; estimating a new third relative change in position ofthe scanning area of the subject by comparing the new first image andthe new second image; calculating a new quality metric of estimation ofthe new second relative change in position based on: the new firstrelative change in position; the new second relative change in position;the new third relative change in position; and comparing the new qualitymetric to the threshold, in the first case wherein the new qualitymetric is less than the threshold, adjusting position of the scanningarea based on the new second relative change in position; and in thesecond case wherein the new quality metric is greater than thethreshold, reobtaining the new second image during a reset new secondperiod of time.

In an alternative embodiment the method may include performingpre-processing on each of the plurality of images before estimatingrelative changes in position; wherein the pre-processing is changedaccording to status of imaging. In an alternative embodiment, thepre-processing may include applying either a smoothing filter or a Sobelfilter according to status of imaging. In an alternative embodiment, thestatus of imaging may be an average intensity of an image among theplurality of images to the pre-processing is to be applied. In analternative embodiment, the status of imaging may be a size of thescanning area. In an alternative embodiment, the status of imaging maybe a focusing position of the imaging light. In an alternativeembodiment, the pre-processing may include dividing each of theplurality of images into smaller images, the estimating of relativechanges in position is performed on the smaller images, and adjustingthe position of the scanning area, are all done during periods of timein which each undivided image is being obtained.

In an alternative embodiment, the method may further comprise:calculating a second quality metric based on a magnitude of the secondrelative change in position; comparing the second quality metric to asecond threshold, in a third case wherein the second quality metric isless than the second threshold, adjusting position of the scanning areabased on the second relative change in position; and in a second casewherein the second quality metric is not less than the second threshold,not adjusting the position of the scanning area.

In an alternative embodiment, the method may further comprise: obtaininga plurality of potential reference images; and setting the referenceimage as an average of the plurality of potential reference images.

In an alternative embodiment, the method may further comprise: obtaininga plurality of potential reference images; and setting one of theplurality of potential reference images as the reference image based onimaging conditions of each of the plurality of potential referenceimages.

In an alternative embodiment, the imaging condition may be an averageintensity of each of the plurality of potential reference images.

In an alternative embodiment, the imaging condition may be a similaritybetween a first potential reference image and a second potentialreference image obtained immediately prior to obtaining the firstpotential reference image.

In an alternative embodiment, in the first case, the second image may beused to construct a larger image by stitching the second image togetherwith other images, and in the second case, the second image is not usedto construct the larger image.

In an alternative embodiment, in the first case, the second image may beused to create a video image by using the second image together withother images to form a time series, and in the second case, the secondimage is not used to construct the video image.

In an alternative embodiment, first image and the second image may beamong a time series of images of the scanning area of the subject. Thesecond image may be the next image after the first image in the timeseries of images of the scanning area of the subject.

An alternative embodiment, may be a non-transitory computer readablemedium encoded with instructions for a processor to perform the methodof an embodiment.

An alternative embodiment, may be a controller including memory and aprocessor. The control may perform the method of an embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate exemplary embodiments.

FIGS. 1A-B are illustrations of ophthalmoscopes in which embodiments maybe implemented.

FIGS. 2A-F is an illustration of images that are divided into stripsthat may be produced in an embodiment.

FIGS. 3A-B are illustrations of methods that may be implemented in anembodiment.

FIG. 4 is an illustration of a method that may be implemented in anembodiment.

FIG. 5 is an illustration of a method that may be implemented in anembodiment.

FIG. 6 is an illustration of a method that may be implemented in anembodiment.

FIGS. 7A-D are illustration of an ophthalmoscopes in which embodimentmay be implemented.

FIGS. 8A-B are illustrations of reference images that may be implementedused in an embodiment.

FIG. 9 is an illustration of a method that may be implemented in anembodiment.

FIG. 10 is an illustration of a controller that may be used in anembodiment.

DESCRIPTION OF THE EMBODIMENTS

Embodiments will be described below with reference to the attacheddrawings. Like numbers refer to like elements throughout. Exemplaryembodiments will be described in detail with reference to the drawingsbelow. It shall be noted that the following description is merelyillustrative and exemplary in nature, and is in no way intended to limitthe disclosure and its applications or uses. The relative arrangement ofcomponents and steps, numerical expressions and numerical values setforth in the embodiments do not limit the scope of the disclosure unlessit is otherwise specifically stated. Techniques, methods, and deviceswhich are well known by individuals skilled in the art may not have beendiscussed in detail since an individual skilled in the art would notneed to know these details to enable the embodiments discussed below.Further, an image photographing apparatus as disclosed in the followingwhich is used to inspect an eye as described below may also be used toinspect other objects including but not limited to skin, and internalorgans.

Ophthalmoscope 1

A first embodiment is described with reference to a fundus imagephotographing apparatus (opthalmoscope) such as the photographingapparatus illustrated in FIG. 1A.

Embodiments are directed towards systems, methods, non-transitorycomputer readable medium, and software which are used in connection withan imaging system such as an opthalmoscope 100. FIG. 1A is anillustration of an exemplary opthalmoscope 100. An opthalmoscope 100 isa system or apparatus for obtaining information about an interiorportion of the eye 111 (e.g., the fundus).

An exemplary embodiment may be a scanning opthalmoscope. A scanningopthalmoscope scans a spot across the eye 111. The spot may be a spot oflight from a light source 101 that is scanned across the eye 111.

In an exemplary embodiment 100, the spot of light is produced by a lightsource 101. The light source 101 may be incorporated into theopthalmoscope 100; alternatively, the opthalmoscope 100 may include aninput for receiving the light source 101. The input for the light source101 may be a fiber optic input or a free space input. The light source101 may be a laser, a broadband light source, or multiple light sources.In an exemplary embodiment, the light source 101 is a super luminescentdiode (SLD) light source having a wavelength of 840 nm. The wavelengthof the light source 101 is not particularly limited, but the wavelengthof the light source 101 for fundus image photographing is suitably setin a range of approximately 800 nm to 1,500 nm in order to reduce glareperceived by a person being inspected while maintaining imagingresolution.

In an exemplary embodiment, light emitted from the light source 101passes through a single-mode optical fiber 102, and is radiated ascollimated light (measuring light 105) by a collimator 103.

In an exemplary embodiment, the polarization of the irradiated light maybe adjusted by a polarization adjusting member 119 (not shown) providedin a path of the single-mode optical fiber 102. In an alternativeconfiguration, the light source 101 is polarized and single-mode opticalfiber 102 is polarization maintaining fiber. In another configuration,the polarization adjusting member may be placed after the collimator103. Alternatively, the polarization adjusting member may be replacedwith a polarizer. In an alternative embodiment, the irradiated light maybe unpolarized, depolarized, or the polarization may be uncontrolled.

The measuring light 105 radiated from the collimator 103 passes througha light division portion 104 including a beam splitter. An exemplaryembodiment includes an adaptive optical system.

The adaptive optical system may include a light division portion 106, awavefront sensor 115, wavefront adjustment device 108, and reflectivemirrors 107-1 to 107-4 for guiding the measuring light 105 to and fromthose components. The reflective mirrors 107-1 to 107-4 are provided toguide the measuring light 105 to and from the pupil of an eye 111, thewavefront sensor 115, and the wavefront adjustment device 108. Thereflective mirrors may be replaced with suitable optics, such as lensesand/or apertures. The wavefront sensor 115 and the wavefront adjustmentdevice 108 may be in an optically conjugate relationship. A beamsplitter may be used as the light division portion 106. The wavefrontsensor 115 may be a Shack-Hartmann sensor or other type of sensor thatgathers information that is representative of the wavefront of lightcoming from the subject.

The measuring light 105 passing through the light division portion 106is reflected by the reflective mirrors 107-1 and 107-2 so as to enterthe wavefront adjustment device 108. The measuring light 105 isreflected by the wavefront adjustment device 108 and is furtherreflected by the reflective mirrors 107-3 and 107-4.

The wavefront adjustment device 108 maybe a transmissive device or areflective device. The wavefront adjustment device 108, may be anaddressable spatial light phase modulator that allows relative phasesacross a beam coming into the wavefront adjustment device 108 to beadjusted such that relative phases across the beam coming out of thewavefront adjustment device 108 are adjustable. In an exemplaryembodiment, one or two spatial phase modulators including a liquidcrystal element is used as the wavefront adjustment device 108. Theliquid crystal element may modulate a phase of only a specific polarizedcomponent. In which case, two liquid crystal elements may be employed tomodulate substantially orthogonal polarized components of the measuringlight 105. In an alternative embodiment, the wavefront adjustment device108 is a deformable mirror.

The measuring light 105 reflected off mirror 107-4 is two-dimensionallyscanned by a scanning optical system 109. In an exemplary embodiment,the scanning optical system 109 includes a first scanner 109-1 and asecond scanner 109-2. The first scanner 109-1 rotates around the firstaxis, while the second scanner 109-2 rotates around a second axis. Thefirst axis is substantially orthogonal to the second axis. Substantiallyin the context of the present disclosure means within the alignment andmeasurement tolerances of the system.

FIG. 1A illustrates the first scanner 109-1 rotating in the x-y plane,while the second scanner 109-2 is rotating in the z-x plane. In thecontext of the present disclosure, rotating the measuring light 105 in afirst plane around the first axis is equivalent to rotating themeasuring light 105 in the first plane and is equivalent to scanning thespot of light in the main scanning direction or the lateral direction ofthe object being imaged. In the context of the present disclosure,rotating the measuring light 105 in a second plane around the secondaxis is equivalent to scanning the spot of light in the sub-scanningdirection or the longitudinal direction of the object being imaged. Thesub-scanning direction is substantially orthogonal to the main scanningdirection.

A scanning period of the first scanner 109-1 is less than the scanningperiod of the second scanner 109-2. The order of the first scanner 109-1and the second scanner 109-2 may be exchanged without impacting theoperation of an exemplary embodiment. The first scanner 109-1 mayoperate in a resonant scanning mode.

In an exemplary embodiment, the scanning optical system 109 may be asingle tip-tilt mirror that is rotated around the first axis and aroundthe second axis that is substantially orthogonal to the first axis. Anexemplary embodiment may also use non-mechanical beam steeringtechniques.

In an exemplary embodiment, the first scanner 109-1 and the secondscanner 109-2 are galvano-scanners. In another exemplary embodiment, oneof the first scanner 109-1 and the second scanner 109-2 is a resonantscanner. The resonant scanner may be used for the main scanningdirection. The resonant scanner may be tuned to oscillate at a specificfrequency. There may be additional optical components, such as lenses,mirrors, apertures, and etc. between the scanners 109-1, 109-2, andother optical components. These additional optical components may bearranged such that the light is focused onto the scanners, in a mannerthat is optically conjugate with all of or one or more of the subject111, the wavefront adjustment device 108, the wavefront sensor 115, anda detector 114.

The measuring light 105 scanned by the scanning optical system 109 isradiated to the eye 111 through eyepieces 110-1 and 110-2. The measuringlight radiated to the eye 111 is reflected, scattered, or absorbed onthe fundus 111. When the eyepieces 110-1 and 110-2 are adjusted inposition, suitable irradiation may be performed in accordance with thediopter of the eye 111. Lenses may be used for the eyepiece portion inthis embodiment, but other optical components such as spherical mirrorsmay also be used.

Light which is produced by reflection, fluorescence, or scattering on afundus of the eye 111 then travels in the reverse direction along thesame path as in the case of incident light. A part of the reflectedlight is reflected by the light division portion 106 to the wavefrontsensor 115 to be used for measuring a light beam wavefront.

In an exemplary embodiment, a Shack-Hartmann sensor is used as thewavefront sensor 115. However, an exemplary embodiment is not limited toa Shack-Hartmann sensor. Another wavefront measurement unit, forexample, a curvature sensor may be employed or a method of obtaining thewavefront by reverse calculation from the spot images may also beemployed.

In FIG. 1A, when the reflected light passes through the light divisionportion 106, a part thereof is reflected on the light division portion104 and is guided to a light intensity sensor 114 through a collimator112 and an optical fiber 113. The light intensity sensor 114 convertsthe light into an electrical signal. The electrical signal is processedby a control unit 117 into an image of the subject, and the image isdisplayed on a display 118.

The wavefront sensor 115 is connected to an adaptive optics control unit116. The received wavefront is transferred to the adaptive opticscontrol unit 116. The wavefront adjustment device 108 is also connectedto the adaptive optics control unit 116 and performs modulation asinstructed by the adaptive optics control unit 116. The adaptive opticscontrol unit 116 calculates a modulation amount (correction amount) toobtain a wavefront having no aberration based on the wavefront obtainedby measuring a result of the wavefront sensor 115, and instructs thewavefront adjustment device 108 to perform the modulation according tothe modulation amount. The wavefront measurement and the instruction tothe wavefront adjustment device are repeated and feedback control isperformed so as to obtain a suitable wavefront.

In an exemplary embodiment the light division portions 104 and/or 106are fused fiber couplers. In an alternative exemplary embodiment, thelight division portions 104 and/or 106 may include partially reflectivemirrors. In another alternative exemplary embodiment, the light divisionportions 104 and/or 106 may include dichroic reflectors, in which case adifferent wavelength of light is used for obtaining an image of thefundus then is used for detecting the spatial phase image that controlsthe adaptive optics system.

The detector 114 may detect reflections or fluorescence associated withthe scanning spot. The detection system may make use confocal microscopytechniques in which an aperture associated with the scanning spot isused to increase the resolution and/or contrast of the detection system.

The adaptive optics system described above includes at least thewavefront sensor 115 and the wavefront adjustment device 108 so that theaberration of the subject's eyes can be measured and compensated for. Adeformable mirror (DM) or a spatial light phase modulator (SLM) can beused as the wavefront adjustment device 108. Since the typical SLM has alarge number of actuators, it can modulate wavefront more precisely thanDM can. A liquid crystal on silicon spatial light modulator (LCOS-SLM)may be used as the wavefront adjustment device 108. The LCOS-SLM 108 canbe controlled to provide a precise spatial modulation of the phase ofthe beam that is used to illuminate the subject.

Opthalmoscope 2

FIG. 1B is an illustration of an alternative embodiment that includessteering mirror 109-3. The steering mirror 109-3 may be a tip-tiltmirror or a pair of scanning mirrors. The steering mirror 109-3 mayinclude galvano, piezo, or mems based mirrors. The steering mirror 109-3is an optical device that steers the position of the optical beam andmay be based upon one or more devices that do not use mirrors such as aspatial phase modulation based steering techniques, lens based steeringtechniques, electro-optical steering, electro-magnetic steering, ormagneto-optical steering.

Multiple Images

The opthalmoscope 100 may take several images and average them togetherto improve the signal/noise ratio and/or contrast ratio of a finalimage. The opthalmoscope 100 may take several images and stitch themtogether to improve the field of view while maintaining the spatialresolution. The opthalmoscope 100 may take several images over period oftime to create a fundus time study. These multi-image studies can bedifficult to produce efficiently because the eye moves continuouslyduring imaging. When the field of view is small such as in AO-SLO theeye movement can be quite large compared with the image size and theimaging area can sometimes go out of frame due to the eye movement. Thiscan increase the amount of time needed to take an image, especially ifmultiple images are required.

Eye Tracking

An eye position tracking system may detect the eye's position with aposition detection device and shift the imaging area according to theeye movement using tracking mirrors.

The tracking mirrors may be implemented using the scanning mirrors 109-1and 109-2. In an alternative embodiment, an additional scanning mirroror scanning mirrors may be used to compensate for the detected changesin eye position.

In an embodiment, the position detection system may be both fast androbust. The position detection system may be an image-based eye trackingprocess for an AO-SLO such as opthalmoscope 100. The process may beimplemented in a real-time AO-SLO system and may work well withhigh-contrast images such as cone mosaic structure of the retina.

In order to compensate for intra-image distortion, a full frame oftarget image 200 is divided into multiple strips 202 (202-1-202-9), asillustrated in FIG. 2A. Whenever a strip 202-1, (e.g., 16 lines) ofimage 200, is digitized, it may be sent to a PC 117 or some otherapparatus that is used for image processing. The PC 117 may include oneor more GPU(s) 120. The one or more GPUs 120 may implement processes forcalculating motion based on the strips 202.

After having received an image strip 202, the one or more GPUs 120perform multiple steps in a process 300 illustrated in FIG. 3A. Themultiple steps may be done sequentially. Some of the steps may be donein parallel. A first step 302 of the process 300 may be an optionalrectification step 302 that corrects sinusoidal distortion caused by theresonant scanner 109-1. The first step 302 may be skipped if a resonantscanner is not used. The rectification step 302 may be done on a line byline basis or on a strip by strip basis.

The process 300 may detect large eye motion. These larger motions may becaused by microsaccades. An estimation of the large eye motion may becalculated using image motion analysis between strips 202-#b from thecurrent image F_(n) 200 -b and a corresponding strip 202-#a from theprevious image F_(n−1) 200 -a, as illustrated in FIG. 2B. For example across-correlation of strip 202-1 b to strip 202-1 a may be used toestimate the motion. FIG. 2B is an illustration of an image F_(n)(200-b) with four strips (202-1 b, 202-2 b, 202-3 b, 202-4 b). Thedetection of microsaccades may be conducted in only the first fewstrips, e.g., 4, instead of the whole image depending on the computationresources available when the microsaccades detection is performed. Thescanning of the rest of the image 200-b may skipped if the motiondetected is greater than a first motion threshold. The first motionthreshold may be based on the ability of the tracking system tocompensate for the motion in time for the rest of the image to beobtained. In an embodiment, the process 300 may include attempting toobtain a new image immediately after the scanning the rest of the imageis stopped or delaying obtaining the new image.

The process 300 may make use of a (global) reference image F₀ (200-c) asillustrated in FIG. 2C. Wherein the subscript “0” identifies thoseimages and strips associated with a global reference image. The dashedline 204-c is representative of an area of the reference imageassociated with the global reference position {right arrow over (R)}₀ asillustrated in FIG. 2D.

FIG. 2D is illustration of the estimated motion of the areas of theimages. The x-y plane is represented as intersecting with the plane ofFIG. 2D in an oblique projection, while time is represented along thehorizontal axis. Dotted lines parallel to the x axis and the y axisillustrate the relative positions of the axes at a time n−1 and a timen.

The process 300 may estimate the eye motion based on a comparison of thestrip S_(n,k) 202 -#b from the current image F_(n) (200-b) and acorresponding strip S_(n,k) 202 -#c from the (global) reference image F₀(200-c). The subscript “n” is an integer that is used to uniquelyidentify those images and strips that are currently being obtained at atime n. Wherein k is an integer used to uniquely identify each strip inthe image. The previously calculated image motion of an image F_(n−1)(200-b) may be used to offset the position of the current strips(202-#b), as illustrated in FIG. 2C. The subscript “n−1” is in referenceto the most recent previous images and strips. The dashed line 204-a isrepresentative of the position {right arrow over (ΔR)}_(n−1), of themost recent image relative the global reference position {right arrowover (R)}₀ as illustrated in FIG. 2D. The estimated motion from eachstrip may be sent to a fast tip/tilt mirror (TTM) to dynamically tracklocation of the eye 111 such as steering mirror 109-3.

The motion of the most recent image {right arrow over (ΔR)}_(n−1), maybe represented by two estimated values along two orthogonal axes(ΔX_(n−1), ΔY_(n−1)). In an alternative embodiment, the estimated valuesfor {right arrow over (ΔR)}_(n−1) may also include additional variablessuch as torsion or depth. The two estimated values (ΔX_(n−1), ΔY_(n−1))may be used for offsetting all of the detected strips 202-#b relative tothe reference image F₀ (200-c). The offset strips which are then used tocalculate the strip motion between the reference image F₀ (200-c) andthe current image F_(n) (200-b). This offset can reduce computationalcosts. The offset may then be added back to each strip to obtain itstrue motion relative to the reference image F₀ (200-c).

When the entire image 200-b has been obtained, after motion from all ofthe strips 200-#b have been estimated, a process that may be implementedon the GPU that calculates the relative image motion {right arrow over(ΔR)}_(n) of the entire image F_(n) (200-c) relative to the referenceimage F₀, which is used to offset positions of strips for the next imageF_(n+1). The image motion {right arrow over (ΔR)}_(n−1) may becalculated on the basis of the entire image or it may be calculated on astrip by strip basis.

As described above the process 300 may include the optional step 302 ofrectification. The rectification step 302 may be done as each line isobtained or on multiple lines at once if it is done a strip by stripbasis. Wherein, the rectification step 302 is performed on a strip k ofthe image F_(n).

The process 300 may also include a step 304 in which the movement of theprevious image F_(n−1) relative to a reference image F₀ is calculated as{right arrow over (ΔR)}_(n−1). In one embodiment, the movement {rightarrow over (ΔR)}_(n−1) is calculated by comparing the entire previousimage F_(n−1) relative to the entire reference image F₀. In anotherembodiment, the movement {right arrow over (ΔR)}_(n−1) may be calculatedby comparing one or more of each strip k in image F_(n−1) to acorresponding strip k in reference image F₀ on a strip by strip basis orby comparing each strip k to the entire reference image F₀ to identifylarge motions. The movement of one or more of the strips k may beestimated and one or more of these estimates may be used to estimate a{right arrow over (ΔR)}_(n−1) final movement of the previous imageF_(n−1) relative to the reference image F₀. The movement may beestimated using cross-correlation imaging tracking techniques or othermethods.

The cross correlation technique may involve calculating the convolutionor correlation of the first image and an offset of the second image asillustrated in equation (1).

$\begin{matrix}{{{A_{n - 1}\left( {\overset{\rightarrow}{B}}_{n - 1} \right)} = {{\left( {F_{0}{\bigstar F}_{n - 1}} \right)\left\lbrack {\overset{\rightarrow}{B}}_{n - 1} \right\rbrack} = {{\left( {F_{0}*F_{n - 1}} \right)\left\lbrack {\overset{\rightarrow}{B}}_{n - 1} \right\rbrack} = {{\sum{F_{0}{{F_{n - 1}\left( {\overset{\rightarrow}{B}}_{n - 1} \right)}\mspace{79mu}\left\lbrack {A_{{n - 1},\max,}{\overset{\rightarrow}{B}}_{{n - 1},\max}} \right\rbrack}}} = {\max\limits_{{\overset{\rightarrow}{B}}_{n - 1}}\left( {A_{n - 1}\left( {\overset{\rightarrow}{B}}_{n - 1} \right)} \right)}}}}}\mspace{79mu} {{\overset{\rightarrow}{\Delta \; R}}_{n - 1} = {\overset{\rightarrow}{B}}_{{n - 1},\max}}} & (1)\end{matrix}$

For real images, the convolution function is equivalent to thecorrelation function. For discrete images, a summation function is usedinstead of an integration function. The summation is performed over thearea of the images F₀ and F_(n−1). The vector {right arrow over(B)}_(n−1) is an arbitrary vector for representing the trial offsets ofimage data F_(n−1) relative to F₀. The function A is representative ofthe results of the correlation function. The function A is evaluated forseveral trial vectors {right arrow over (B)}_(n−1) and the maximum valuefor the function A is sought out. {right arrow over (ΔR)}_(n−1) is thatvector {right arrow over (B)}_(n−1) at the maximum value of the functionA. A peak search method may also be used to refine the value {rightarrow over (ΔR)}_(n−1). Other methods may be used to find {right arrowover (ΔR)}_(n−1) using some quantitative method of comparing thereference image F₀ and the offset image F_(n−1).

The process 300 includes a step 306 of offsetting each strip k of theimage F_(n) based upon the estimated motion {right arrow over(ΔR)}_(n−1) of the previous image as illustrated in FIG. 2C. The processmay also include an estimation step 308 in which each offset strip k iscompared to a corresponding strip k in reference image F₀ to obtain anestimation of movement {right arrow over (ΔR)}_(n,k) of the strip k inimage F_(n) relative to the reference image F₀ which is correlated withthe movement of the subject 111 relative to the reference position. Theestimation step 308 may be performed by using cross-correlation and/orother image based motion estimation/tracking methods. The process 300may include a compensation step 310 in which estimated movement {rightarrow over (ΔR)}_(n,k) of the strip k in image F_(n) relative to thereference image F₀ is then used to reposition the scanning area in thesubsequent strip k+1 or a later strip k+x in which x is greater than 1and depends on the processing time. The offset {right arrow over(ΔR)}_(n−1) may be removed to give a measure of the current motionrelative to the most recent image. The compensation step 310 may includeadjusting the scanners 109-1 and 109-2. The compensation step 310 mayinstead make use of an additional tip/tilt scanner 109-3 or set ofscanners that are used specifically for positioning the scanning areainstead of the scanners which are used for scanning the measurementbeam.

Robust Tracking Method

As discussed above we may define strip motion between the referenceimage F₀ and the target image F_(n) with a vector {right arrow over(ΔR)}_(n,k) (ΔX_(n,k), ΔY_(n,k)) where n is an image index of the targetimage F_(n) and k is strip index from the target strip S_(n,k), anddefine strip motion between the two corresponding strips (S_(n,k),S_(n−1,k)), in two consecutive images, target image F_(n) and itsprevious image F_(n−1), as {right arrow over (δR)}_(n,k) (δX_(n,k),δY_(n,k)). The applicants have found that it is useful to verify theself-consistency of the estimated motion with the followingapproximation equation (2a). Where {right arrow over (ΔR)}_(n−1,k)(ΔX_(n−1,k), ΔY_(n−1,k)) is strip motion between image F_(n−1) and thereference image F₀. Equation (2b) is reformulation of equation (2a) tocome up with a quality metric Q_(n,k) as a comparison test. In equation(2b) illustrates how the vector {right arrow over (ΔR)}_(n−1,k) is addedto the vector {right arrow over (δR)}_(n,k) the vector {right arrow over(ΔR)}_(n,k) is then subtracted from this vector sum. The applicants havefound that magnitude Q_(n,k) of this calculation can be used asestimation of the quality of the tracking estimation. The quality metricQ_(n,k) may then be compared to a threshold R_(max). If the qualitymetric Q_(n,k) is less than a threshold R_(max) as described equation(2c) then eye tracking estimation may be considered good enough forcontrolling the motion of a tracking mirror.

{right arrow over (ΔR)}_(n,k)≈{right arrow over (ΔR)}_(n−1,k)+{rightarrow over (δR)}_(n,k)  (2a)

Q _(n,k)=|{right arrow over (ΔR)}_(n−1,k)+{right arrow over(δR)}_(n,k)−{right arrow over (δR)}_(n,k)  (2b)

R _(max) >Q _(n,k)  (2c)

The quality metric Q_(n,k) is representative of the quality of: eyetracking estimation of the current image, eye tracking estimation of theprevious image, the current image; and the previous image.

FIG. 2E is an illustration of the how the different images may be usedto estimate the target strip motion {right arrow over (ΔR)}_(n,k) withdata from strips between two consecutive images {right arrow over(δR)}_(n,k).

Due to the fluctuation of image brightness and contrast, noise level,and possible eye torsion, calculation of the target strip motion {rightarrow over (ΔR)}_(n,k) between the target image F_(n) and the referenceimage F₀ as time intervals increase to seconds or tens of seconds,becomes less reliable. The calculation can become less reliable as thecross correlation method can provide a false positive by finding a falsepeak location. However, the estimation of {right arrow over (δR)}_(n,k)from two consecutive images (F_(n), F_(n−1)) is substantially morereliable because the time interval can be on the order of tens ofmilliseconds, instead of seconds. Thus fluctuations in image brightness,contrast, noise level, and eye torsion have significantly less negativeimpact on the calculation of {right arrow over (δR)}_(n,k). Therefore,{right arrow over (δR)}_(n,k) may be used to further estimate {rightarrow over (ΔR)}_(n,k) where {right arrow over (ΔR)}_(n,k) may bereplaced by {right arrow over (ΔR)}_(n−1,k)+{right arrow over(δR)}_(n,k) if the cross correlation returns an unreasonable {rightarrow over (ΔR)}_(n,k).

FIG. 3B is an illustration of a method 301 which may implement therobust tracking method which may be used in an embodiment. The method301 may include a step 303 of setting an image as a reference image F₀.The method 301 may include a step 305 of setting an image from among aplurality of images as a first image F_(n−1). The method 301 may includea step 307 of setting an image from among a plurality of images as thesecond image F_(n−1). The second image may be an image obtainedimmediately after the first image is obtained. The second image may bean image of the same area as the first image obtained most recentlyafter the first image was obtained that also meets certain criteria,such as lack of interference from an eye lash or a blink.

The method 301 may include the step 304 as used in method 300 in whichthe movement of the first image F_(n−1) relative to a reference image F₀is estimated as {right arrow over (ΔR)}_(n−1). The method 301 mayinclude a step 309 in which the movement of the second image F_(n)relative to a reference image F₀ is estimated as {right arrow over(ΔR)}_(n). The method 301 may include a step 311 in which the movementof the second image F_(n) relative to the first image F_(n−1) isestimated as {right arrow over (δR)}_(n).

The method 301 may include a step 313 of calculating a quality metricQ_(n) such as using equation (2c) based on {right arrow over(ΔR)}_(n−1), {right arrow over (ΔR)}_(n), and {right arrow over(δR)}_(n). The method 301 may include a step 315 of comparing thequality metric Q_(n) to a threshold R_(max). In a first case wherein thequality metric meets threshold criteria the method may include a step317 in which the imaging position is adjusted based on the estimatedrelative change in position {right arrow over (ΔR)}_(n). The step 317may also include incrementing index n, in which case the second imagebecomes the first image, a new second image, and estimation andcalculation steps are calculated for the new index n. In a second casein which the quality metric does not meet the threshold then a newsecond image is obtained in step 307. In which case new estimates andmetrics are calculated and compared. The method 301 may be donerepeatedly forming a feedback loop and may be done for full images orfor sub-images such as strips.

Pre-Processing of Images

When obtaining low-contrast images, including vascular images,additional pre-processing of raw images may be employed beforecross-correlation is performed. The pre-processing may start with a2-dimensional smoothing filter, followed by a 2-dimensional Sobelfilter, and followed by thresholding to remove filtering artifacts.Filter combinations may be switched according to image signal strength,FOV size and focusing position.

FIG. 4 is an illustration of a pre-processing method 400 describedabove. The pre-processing step may be performed before or after theoptional rectification step 302. The pre-processing method 400 may beperformed on a strip by strip basis or on a line by line basis to theimage data obtained by the detector 114. The pre-processing method 400may be performed by one or more of a PC 117, a GPU 120, and/or dedicatedcircuitry.

The pre-processing method 400 may include an optional firstpre-processing step 412 of applying a smoothing filter to the image dataobtained by the detector 114 to obtain smoothed image data. Thesmoothing filter in the first pre-processing step 412 may be atwo-dimensional smoothing filter. Applying the smoothing filter mayinclude convolving a filter function with the image data. The smoothingfilter may be a single pass or a multi-pass filter. One or moredifferent kinds of filter functions may be used including Gaussianfilters, hat filters, adaptive filters, and other filter functions.Other methods of applying a smoothing filter may also be used all ofwhich include methods of reducing noise in the input image data with theend results of producing smoothed image data with less noise.

The pre-processing method 400 may include an optional secondpre-processing step 414 of applying a Sobel filter to the smoothed imagedata or the image data from the detector 114. Applying the Sobel filtermay include convolving data with a kernel, the kernel may be a Sobelkernel or other similar kernels.

The pre-processing method 400 may include an optional thirdpre-processing step 416 of applying a thresholding step. Thethresholding step 416 may include applying a specific threshold or anadaptive threshold to data produced in the previous pre-processingsteps. The threshold step 416 may be used remove artifacts produced bythe previous filtering steps.

Adaptive Strip Size

In order to improve upon the quality of low-contrast images, includingvascular images, a size of strips may be changed as illustrated in anadaptive strip size method 500 illustrated in FIG. 5. The inventors havefound that with larger strips the accuracy and robustness ofcross-correlation is higher. Although, when the strip size is increasedthen the time that the imaging system needs to respond to the subjects,motion is decreased. In one alternative embodiment, the number of linesof image data used in a strip to perform cross-correlation so as toestimate the position of the subjected being is changed (increased ordecreased) while the image is being obtained.

A first step 518 in the adaptive strip size method 500 may bedetermining the status of the current image as it is being obtained. Thestatus of the image may include one or more of: magnification;resolution; spot size; focal point; relative imaging position on thesubject's fundus; light intensity; wavelength; statistics associatedwith image properties such as image signal strength and contrast ratio;higher order image data statistics such as the distribution and numberof distinct features that can be identified in the imaging area of thestrip; measures of the reliability of the cross-correlation estimationas used in the particular strip or a recent strip; peak value of thecross-correlation function in the current particular strip or a previousparticular strip; the width of the peak; and/or the amount of positionshift detected. One or more of these values may then be compared to oneor more thresholds. The results of these comparisons can give anestimation of the robustness of the current strip and its ability togive a quality estimate of the motion of the subject.

A second step 520 in the adaptive strip size method 500 may be changingthe size of the strip based upon the status of the present imagingmethod. The number of lines used in the current strip may be increased,decreased, or kept the same based upon the determination made in step518. For instance, if the status of the present imaging method isindicative that the estimation in the change of movement as detectedusing the current strip size will be less than a particular lower limitthreshold then the number lines included in the particular strip may beincreased, thus improving the reliability of the position estimationmethod. Likewise, if the status of the present imaging method isindicative that the estimation in the change of movement as detectedusing the current strip size will be greater than a particular upperlimit threshold then the number lines included in the particular stripcan be decreased, thus increasing the speed with which position changesare detected and compensated for.

Microsaccade Detection

In an integrated system where a wide field of view scanning laserophthalmoscope (WFOV-SLO) and an adaptive optics scanning laserophthalmoscope (AO-SLO) do eye tracking concurrently, the WFOV-SLO isusually able to detect microsaccades more accurately than the AO-SLOdoes because of its large scanning FOV. The AO-SLO tracking may benotified dynamically by the WFOV-SLO tracking software when amicrosaccade occurs so that AO-SLO tracking software may freeze a tiptilt mirror 109-3 without doing additional computation.

Automatic Reference Image Selection Method

An imaged-based eye tracking process such as the exemplary process 300illustrated in FIG. 3A makes use of a reference image 204-C F₀. Thereference image 204-C F₀ may be determined manually. In which case anoperator chooses a “good” image from a live video stream and analgorithm then drives a scanning system such which may include a tiptilt mirror 109-3 to stabilize all the images following images withrespect to the chosen reference image 204-C F₀. In real-time humanretinal imaging, it can be difficult to predict the occurrence of ablink or a microsaccade. Thus, it is not unusual for an operator tochoose a reference image 204-C F₀ from a highly distorted microsaccadeinterval, or from a blink interval in which the reference image 204-C F₀is of poor quality in the context of having suitable features to be usedin the cross correlation calculation. When this situation arises, oftenthe best solution is to stop tracking, reset, and use a differentreference image 204-C F₀. When a patient's eye is unstable, moving toofast, the occurrence of microsaccades or blinks is too frequent,manually choosing a good reference image 204-C F₀ can become a problemfor an operator. It can be challenging for an operator to manuallycapture a reference image 204-C F₀ from the live video feed.

A reference image selection method 600 illustrated in FIG. 6 can choosean appropriate reference image for the operator. The reference imageselection method 600 may start with when operator pushes a software orhardware button “start tracking”. In an embodiment, a processor 764 mayreceive a request to start tracking in a step 622 of method 600. Thecriteria for any random reference image for being a ‘good’ referenceimage may be based on data obtained from the two consecutive images,i.e., {right arrow over (δR)}_(n,k) as illustrated in FIG. 2F. In whichcase, in a calculation step 624 the processor 764 may calculate {rightarrow over (δR)}_(n,k) for one or more strips. {right arrow over(δR)}_(n,k) may be calculated using the same method described inequation (1) shown here as equation (3). Equation (3) is substantiallysimilar to equation (1). In which each strip S_(n,k) is indexed at atime index n, and a strip index k. The variable C_(n,k) isrepresentative of the results of the cross correlation function for eachstrip at time index n and strip number k.

$\begin{matrix}{{{C_{n,k}\left( {\overset{\rightarrow}{B}}_{n,k} \right)} = {{\left( {S_{{n - 1},k}{\bigstar S}_{n,k}} \right)\left\lbrack {\overset{\rightarrow}{B}}_{n,k} \right\rbrack} = {{\left( {S_{{n - 1},k}*S_{n,k}} \right)\left\lbrack {\overset{\rightarrow}{B}}_{n,k} \right\rbrack} = {{\sum{S_{{n - 1},k}{{S_{n,k}\left( {\overset{\rightarrow}{B}}_{n,k} \right)}\mspace{79mu}\left\lbrack {C_{n,k,\max},{\overset{\rightarrow}{B}}_{n,k,\max}} \right\rbrack}}} = {\max\limits_{{\overset{\rightarrow}{B}}_{n - 1}}\left( {C_{n,k}\left( {\overset{\rightarrow}{B}}_{n,k} \right)} \right)}}}}}\mspace{79mu} {{\overset{\rightarrow}{\delta \; R}}_{n,k} = {\overset{\rightarrow}{B}}_{n,{k.},\max}}} & (3)\end{matrix}$

The reference image selection method 600 may include a comparison step626. The comparison step 626 may include comparing the results of stripmotion calculation to a threshold. In one embodiment, {right arrow over(δR)}_(n,k) from a set of strips are compared to a threshold. In oneembodiment, the set of strips may include all of the strips that make upa particular image. In one alternative embodiment, the set of strips mayinclude all of the strips that make up multiple images. In oneembodiment, C_(n,k,max) from a set of strips among one or more imagesmay be compared to a threshold. In another embodiment, statisticsassociated with the C_(n,k,max) calculation, such as a peak widthassociated with C_(n,k,max), from a set of strips among one or moreimages may be compared to a threshold. Multiple different criteria maybe compared to multiple different thresholds during the comparison step626.

The reference image selection method 600 includes an image selectionstep 628, wherein, the result of the comparison done in step 626 mayidentify a pair of images F_(n) and F_(n−1). The identified images arethose images that pass the comparison test. Either of these images F_(n)and F_(n−1) may then be used as a reference image 204-c F₀. In a firstcase, the threshold test identifies an image in which the estimatedmovement {right arrow over (δR)}_(n,k) is less than a threshold for allof k in an image n then the images F_(n) or F_(n−1) may be consideredgood images and used as the reference image 204-c F₀. In a second case,the threshold test identifies an image in which correlation coefficientsfrom a set of strips are higher than a used-defined threshold, then theimages F_(n) or F_(n−1) may be considered good images and used as thereference image 204-c F₀.

Method for an Embodiment that Includes a WF-SLO and an AO-SLO

An embodiment may be implemented in an integrated ophthalmoscope 700-1as illustrated in FIGS. 7A-C. An embodiment may also be a method that isused in the context of the integrated ophthalmoscope 700-1.

The integrated ophthalmoscope 700-1 may be also referred to as thesystem 700-1 in the context of this application. The ophthalmoscope700-1 includes an ocular lens unit 730, a wide field scanning laserophthalmoscope (WF-SLO) 732, an adaptive optics scanning laserophthalmoscope (AO-SLO) 734, and an internal fixation target 736. Ascanning beam of the WF-SLO 732 and a scanning beam of the AO-SLO 734illuminate a fundus of a subject 111. In one embodiment, the beams enterthe fundus simultaneously, and each of the beams may have differentwavelengths. In another embodiment, the beams enter the fundus atseparate times. This may be done using pulsed light sources, opticalswitches, a single light source and a dual output Mach-Zehndermodulator, or a variety of other methods in which light can be combinedand then independently detected may be used.

The ocular lens unit 730 may include a first beam combiner 738 forcombining the beams to and from the WF-SLO 732 and the beams to and fromthe AO-SLO 734 to form a combined beam. The first beam combiner 738 mayinclude a partially silvered mirror in which case, the beams may havesimilar wavelengths. The first beam combiner 738 may include a dichroicmirror in which case, wavelength ranges of the beams to and from theWF-SLO 732 may be different from wavelength ranges of the beams to andfrom the AO-SLO 734. The first beam combiner 738 may be a switch inwhich case the switch may be used to temporally multiplex beams to andfrom the WF-SLO 732 and the beams to and from the AO-SLO 734.

The ocular lens unit 730 may also include a first eyepiece or lens 110-1and a second eyepiece or lens 110-2 which are used to transmit thecombined light from the beam combiner 738 to an object 111. The firsteyepiece 110-1 and the second eyepiece 110-2 may be replaced with curvedmirrors. A dichroic filter 740 may be inserted between the firsteyepiece 110-1 and the second eyepiece 110-2 in order to combine thecombined light from the beam combiner 738 with light from an internalfixation target 736. In an alternative embodiment, the dichroic filter740 may be inserted between the second lens 110-2 and the object 111.The internal fixation target 736 may include a first light source 742and a third lens 744. The first light source 742 may be multiple lightemitting diodes arranged in matrix. A turn-on position of the lightemitting diodes in the first light source 742 may be changed by acontroller 116 in accordance with the part of the eye 111 desired to beimaged. Light from the first light source 742 is guided to the eye 111to be inspected by the dichroic filter 740 via the third lens 744. Thelight emitted from the first light source 742 is visible light and mayhave a wavelength of 520 nm.

An example of the WF-SLO 732 is illustrated in FIG. 7B. The WF-SLO 732includes a second light source 746 or an input for receiving light fromthe second light source 746. The second light source 746 may be a laser,a lamp, a diode, a semiconductor laser, a super luminescent diode, a gaslaser, a fiber coupled light source, a fiber based light source, someother suitable light source, or a set of light sources. The second lightsource 746 may be coherent or incoherent. In order to reduce thebrightness as perceived by the subject while maintaining suitableresolution for fundus observation, a wavelength of the second lightsource 746 may be in the near infrared range of 700 nm to 1,000 nm. Thewavelength of the second light source 746 may be non-visible light. Inan embodiment, a semiconductor laser having a wavelength of 780 nm isused. Light emitted from the second light source 746 may be transmittedthrough a fiber to be emitted from a fiber collimator as a collimatedbeam (measuring light).

The measuring light passes through a first beam splitter 750 and afourth lens 748-1 and is guided to a first scanner 109-4. The firstscanner 109-4 may be a resonant scanner which scans the light along afirst axis. The measuring light from the first scanner 109-4 may thenpass through a fifth lens 748-3 and a sixth lens 748-4 which may focusthe light onto a second scanner 109-5. The second scanner 109-5 may be agalvano scanner driven by a triangle signal or a saw tooth signal bothof which are ramp signals along a second axis. Light from the secondscanner is then transmitted via the ocular lens unit 730 to the subject111. The ocular lens unit 730 gathers light from the subject 111 andguides back into the WF-SLO 732. The gathered light from the subject 111that has been guided back to WF-SLO 732 by the combiner 738 is then sentback through the second scanner 109-5, the fifth lens 748-3 and thesixth lens 748-4, and the first scanner 109-4 before it hits the firstbeam splitter 750. The first beam splitter 750 guides a portion of thegathered light to a seventh lens 748-2 and to a first detector 752.

The first axis and the second axis are substantially orthogonal to eachother. The first axis corresponds to a main scanning direction, and thesecond axis corresponds to a sub scanning direction.

The beam that has entered the eye 111 irradiates a fundus of the eye toinspect the subject 111 using the spot formed by the beam. This beam isreflected or scattered by the fundus of the eye 111 and follows the sameoptical path and is received by the first detector 752 which may be anavalanche photodiode (APD).

In an alternative embodiment, the second source 746 is a fiber coupledlight source, the seventh lens 748-2 is GRIN lens that couples the fibercoupled light to the first detector 752, the first beam splitter 750 isa fused fiber coupler or a fiber coupled beam splitter, and the fourthlens 748-1 is a fiber coupled GRIN lens.

FIG. 7C is an illustration of an exemplary AO-SLO 734 which may be usedin an embodiment. The AO-SLO 734 may use a third light source 101. Thethird light source 101 may be a SLD light source having a wavelength of840 nm. The third light source 101 may be a non-visible light source oran infrared light source. The third light source 101 may have wavelengthspectrum that is different from a wavelength spectrum of the secondlight source 746. The third light source 101 may be shared for imagingthe fundus and for measuring a wavefront of the image. In an alternativeembodiment, the wavefront may be measured with one wavelength while thefundus is imaged with a second different wavelength.

The light emitted from the third light source 101 may be transmittedthrough a fiber to be radiated via an eighth lens 103 or a fibercollimator as collimated measuring light. The embodiment may include thethird light source 101 or include an input for the third light source101. The radiated measuring light is transmitted through a second beamsplitter 104 and guided to a compensation optical system.

The compensation optical system includes a third beam splitter 106, awavefront sensor 115 for measuring aberration, a wavefront correctiondevice 108, and mirrors 107-1 to 107-4 (first mirror 107-1; secondmirror 107-2; third mirror 107-3; and fourth mirror 107-4) for guidingthe light to those components. The mirrors 107-1 to 107-4 may bearranged such that the eye 111 to be inspected, the wavefront sensor115, and the wavefront correction device 108 may have an opticallyconjugate relationship to each other. The wavefront correction device108 may be a spatial phase modulator, such as a reflective liquidcrystal element, a deformable mirror, or a translucent liquid crystalelement.

The measuring light may enter the wavefront correction device 108 andmay be emitted to the third mirror 107-3. Similarly, the light that hasreturned from the fundus of the eye 111, enters the wavefront correctiondevice 108, and is then emitted to the second mirror 107-2. Themeasuring light is scanned two-dimensionally by a third scanner 109-1which is then sent to a fourth scanner 109-2 via a ninth lens 754 and atenth lens 756. The third scanner 109-1 may be a high-speed resonancescanner. The fourth scanner 109-2 may be a galvano scanner. The thirdscanner 109-1 and the fourth scanner 109-2 scan in directions that arein substantially orthogonal directions to each other. The third scanner109-1 may scan along the first axis in the same manner as the firstscanner 109-4. The third scanner 109-1 may scan along the x-axis. Thefourth scanner 109-2 may scan along the y-axis.

The measuring light scanned by the third scanner 109-1 and the fourthscanner 109-2 is then transmitted via the ocular lens unit 730 to thesubject 111. The ocular lens unit 730 gathers light from the subject 111and guides it back into the AO-SLO 734. The gathered light from thesubject 111 that has been guided back to AO-SLO 734 by the combiner 738is then sent back through the fourth scanner 109-2, the ninth lens 754and the tenth lens 756, the third scanner 109-1, mirror 107-4, mirror107-3, wavefront correction device 108, mirror 107-2, and mirror 107-1,before it hits the third beam splitter 106. The third beam splitter 106guides a portion of the gathered light to the wavefront sensor 115. Thethird beam splitter 106 guides another portion of the gathered light toa second beam splitter 104. The second beam splitter 104 guides thegathered light to an eleventh lens 112 and to a second detector 114.

The measuring light that has entered the eye 111 to be inspected hasbeen reflected or scattered by the fundus and follows the same opticalpath as the measurement light. The wavefront sensor 115 measures awavefront of the beam from the fundus. The wavefront sensor 115 may be aShack-Hartmann sensor, a pyramid wavefront sensor, or some other methodfor characterizing a wavefront. The detector 114 may be aphotomultiplier tube. The detector 114 may be a fiber coupled detectorand the lens 112 may be a fiber collimator. In an alternativeembodiment, the second beam splitter 104 may be a partially silveredprism.

In an alternative embodiment, the third source 101 is a fiber coupledlight source or is coupled to a fiber via the eighth lens 103, theeleventh lens 112 is a GRIN lens that couples the fiber coupled light tothe second detector 114, the second beam splitter 104 is a fused fibercoupler or a fiber coupled beam splitter, and is collimated by a fibercoupled GRIN lens.

The first detector 752 converts light into an electrical signal which isthen converted into a wide field image of a portion the fundus Ea. Thesecond detector 114 converts light into an electrical signal which isthen converted into a narrow field image of a narrower portion of thefundus.

The wavefront sensor 115 and the wavefront correction device 108 areconnected to each other. One or more processors associated with thecontroller 116 and/or the PC 117 may be used to calculate a modulationamount of the wavefront correction device 108 which is calculated basedon information from the wavefront sensor 115. The wavefront sensor 115and the wavefront correction device 108 may form a closed loop feedbacksystem.

FIG. 7D is an illustration of an embodiment 700-2 substantially similarto embodiment 700-1 and also includes an additional tracking mirror109-3. The embodiment 700-2 may also include an optional relay lenses110-3 and 110-4. The tracking mirror 109-3 may consist of 2 scanners orone 2-dimensional scanner. The tracking mirror 109-3 is moved tocompensate for detected movements of the subject 111, such that thefundus stays in focus and within the field of view. In an alternativeembodiment, there is no tracking mirror and the scanning mirrors 109-4,109-5, 109-1, and 109-2 are moved to compensate for the detectedmovements of the subject 111, such that the fundus stays in focus andwithin the field of view. In other embodiments, there may be additionaltracking mirrors to compensate for the detected movements of the subject111 in addition to the scanning mirrors 109-4, 109-5, 109-1, and 109-2.

An alternative embodiment, may include multiple different tip-tiltmirrors or two sets of single axis mirrors may be used in concert foroptical stabilization. A coarse tip-tilt mirror may be used for coarseoptical stabilization. A fine tip-tilt mirror may be used for fineoptical stabilization. One or both of the tip-tilt mirrors may bereplaced with two single axis rotating mirrors.

In an integrated ophthalmoscope such as 700-1 or 700-2 where a WF-SLO732 and an AO-SLO 734 do eye tracking concurrently, it may be possibleto lock a steering mirror such as 109-3 or its equivalent to the sameretinal area from the WF-SLO 732 from the same subject 111. This featuremay require the WF-SLO 732 to use the same reference image for allimaging sessions. The applicants have noted that images produced by theWF-SLO 732 typically have less intra-image distortion than imagesproduced by the AO-SLO 734. different reference images produced by theWF-SLO 732 may be pre-recorded from imaging sessions at differentretinal locations. FIG. 8A is an illustration of a reference image 804-c1 that may be produced by the WF-SLO 732. A particular reference image804-c 1 may be chosen by an operator or automatically picked from asequence of images. An alternative reference image 804-c 2 asillustrated in FIG. 8B may be produced by averaging multiple images.

Different reference images at different retinal locations on the subject111 may be stored in volatile memory or non-volatile memory such as ahard drive or a flash drive. The reference images may be stored, on theophthalmoscope, a locally connected computer, or a storage device thatis accessed via a network. One of the reference images may be loadedfrom a storage location and into a local memory used by an embodimentthat performs eye tracking. The eye tracking may be performed bytracking software. The reference image may be loaded into memory beforethe eye tracking software is launched. A different reference image froma different retinal area may be loaded into the tracking software whenthe imaging area is moved to a different retinal area.

First Alternative Automatic Reference Image Selection Method

FIG. 9 is an illustration of a method 900 for choosing a referenceimage. A first step 958 may include obtaining a plurality of potentialreference images. A second step 960 may include calculating a metricbased on a signal strength associated with each of the potentialreference images. The metric may include the average intensity of eachpotential reference image. A third step 962 may include choosing thereference image based on the metric of each potential reference image.For example, the particular reference image may be chosen when themetric for the particular reference image is in a range of values. Themetric may also be based on identifying a particular image structurewithin the image. Examples, of particular image structures, may includethe clear appearance of branches of vessels, thick vessels, or opticnerve heads.

The tracking method may use reference images that were pre-recorded atan earlier date. The AO-SLO and the WF-SLO may use different trackingmethods.

Controller

FIG. 10 is an illustration of the PC 116 and controller 117 that may beused in an embodiment. The controller 116 receives input signals andoutputs control signals. The controller 116 may be a general purposecomputer, a device specifically designed to control the ophthalmoscopeor measuring instrument, or a hybrid device that uses some customelectronics along with a general purpose computer 117. The input signalsand control signals may be digital signals and/or analog signals. Thecontroller 116 may include an analog to digital converter (ADC) and adigital to analog converter (DAC). The input signals may include onemore signals such as a signal from the wavefront sensor 115, a signalfrom the detector 114, and one or more signals from one or more othersensors. The control signals may include a first control signal to awavefront adjustment device 108 and signals to one or more of thescanners 109-1, 109-2, and/or 109-3. The control signals may includeadditional signals to other components of the instrument.

The controller 116 includes a processor 764-1. The processor 764-1 maybe a microprocessor, a CPU, an ASIC, a DSP, and/or a FPGA. The processor764-1 may refer to one or more processors that act together to obtain adesired result. The controller 116 may include a memory 766-1. Thememory 766-1 may store calibration information. The memory 766-1 mayalso store software for controlling the ophthalmoscope. The memory 766-1may be a form of a non-transitory computer readable storage medium. Thenon-transitory computer readable storage medium may include, forexample, one or more of a hard disk, a random-access memory (RAM), aread only memory (ROM), a distributed storage system, an optical disk(CD, DVD or Blu-Ray Disc, a flash memory device, a memory card, aninternet connected storage device, an intranet connected storage device,or the like.

The controller 116 may be connected to a computer (PC) 117 via a directconnection, a bus, or via a network. The computer 117 may include inputdevices such as a keyboard, a mouse, and/or a touch screen. Thecontroller 116 may include input devices such as a keyboard, a mouse, atouch screen, knobs, switches, and/or buttons. The computer 117 may beconnected to a display 118. The results of the ophthalmoscope may bepresented to a user via the display 118. The tracking software which maybe used to implement an embodiment that may perform calculations on thecontroller 116 independently of the PC 117 or with the help of the PC117. The PC 117 may include a processor 764-2, a memory 766-2. The PC117 may also include one or more GPUs 120. The image correlationcalculations may be performed on the one or more GPUs 120.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all modifications, equivalent structures, and functions.

What is claimed is:
 1. A method of imaging a scanning area of a subjectby scanning imaging light over periods of time on the scanning area ofthe subject, the method comprising: constructing a plurality of imagesbased on detection of the imaging light from the scanning area of thesubject, each image among the plurality of images obtained during adifferent time period; wherein the plurality of images includes atleast: a reference image, a first image, and a second image; wherein thefirst image is obtained during a first period of time, the second imageis obtained during a second period of time, the reference image isobtained during a third period of time that is prior to the first periodof time and the second period of time; estimating a first relativechange in position of the scanning area of the subject by comparing thereference image and the first image; estimating a second relative changein position of the scanning area of the subject by comparing thereference image and the second image; estimating a third relative changein position of the scanning area of the subject by comparing the firstimage and the second image; calculating a quality metric of estimationof the second relative change in position based on: the first relativechange in position; the second relative change in position; the thirdrelative change in position; and comparing the quality metric to athreshold, in a first case wherein the quality metric is less than thethreshold, adjusting position of the scanning area based on the secondrelative change in position; and in a second case wherein the qualitymetric is not less than the threshold, obtaining a new second imageduring a new second period of time.
 2. The method of claim 1, wherein inthe second case, the position of the scanning area is controlled basedupon previous image position information.
 3. The method of claim 1,wherein the second image is an image of the scanning area of the subjectthat is obtained immediately after the first image is obtained.
 4. Themethod of claim 1, further comprising in the first case: setting thesecond image as a new first image; setting the second relative change inposition as a new first relative change in position; obtaining the newsecond image during the new second period of time, wherein the newsecond image is an image of the scanning area of the subject that isobtained immediately after the new first image is obtained; estimating anew second relative change in position of the scanning area of thesubject by comparing the reference image and the new second image;estimating a new third relative change in position of the scanning areaof the subject by comparing the new first image and the new secondimage; calculating a new quality metric of estimation of the new secondrelative change in position based on: the new first relative change inposition; the new second relative change in position; the new thirdrelative change in position; and comparing the new quality metric to thethreshold, in the first case wherein the new quality metric is less thanthe threshold, adjusting position of the scanning area based on the newsecond relative change in position; and in the second case wherein thenew quality metric is greater than the threshold, reobtaining the newsecond image during a reset new second period of time.
 5. The method ofclaim 1 further comprising: performing pre-processing on each of theplurality of images before estimating relative changes in position; andwherein the pre-processing is changed according to status of imaging. 6.The method of claim 5, wherein the pre-processing includes applyingeither a smoothing filter or a Sobel filter according to status ofimaging.
 7. The method of claim 5 wherein the status of imaging is anaverage intensity of an image among the plurality of images to thepre-processing is to be applied.
 8. The method of claim 5 wherein thestatus of imaging is a size of the scanning area.
 9. The method of claim5, wherein the status of imaging is a focusing position of the imaginglight.
 10. The method of claim 5, wherein pre-processing includesdividing each of the plurality of images into smaller images, theestimating of relative changes in position is performed on the smallerimages, and adjusting the position of the scanning area, are all doneduring periods of time in which each undivided image is being obtained.11. The method of claim 1, further comprising: calculating a secondquality metric based on a magnitude of the second relative change inposition; comparing the second quality metric to a second threshold, ina third case wherein the second quality metric is less than the secondthreshold, adjusting position of the scanning area based on the secondrelative change in position; and in a second case wherein the secondquality metric is not less than the second threshold, not adjusting theposition of the scanning area.
 12. The method of claim 1 furthercomprising: obtaining a plurality of potential reference images; andsetting the reference image as an average of the plurality of potentialreference images.
 13. The method of claim 1 further comprising:obtaining a plurality of potential reference images; and setting one ofthe plurality of potential reference images as the reference image basedon imaging conditions of each of the plurality of potential referenceimages.
 14. The method of claim 13, wherein the imaging condition is anaverage intensity of each of the plurality of potential referenceimages.
 15. The method of claim 13, wherein the imaging condition is asimilarity between a first potential reference image and a secondpotential reference image obtained immediately prior to obtaining thefirst potential reference image.
 16. The method of claim 1 wherein, inthe first case, the second image is used to construct a larger image bystitching the second image together with other images, and in the secondcase, the second image is not used to construct the larger image. 17.The method of claim 1 wherein, in the first case, the second image isused to create a video image by using the second image together withother images to form a time series, and in the second case, the secondimage is not used to construct the video image.
 18. The method of claim1, wherein: first image and the second image are among a time series ofimages of the scanning area of the subject; and the second image is thenext image after the first image in the time series of images of thescanning area of the subject.
 19. A non-transitory computer readablemedium encoded with instructions for a processor to perform the methodof claim
 1. 20. A controller for controlling an apparatus for imaging ascanning area of a subject by scanning imaging light over periods oftime on the scanning area of the subject, comprising: a processorincluding one or more processing components; a memory; wherein theprocessor constructs a plurality of images based on detection of theimaging light from the scanning area of the subject, each image amongthe plurality of images obtained during a different time period; whereinthe plurality of images includes at least: a reference image, a firstimage, and a second image; wherein the first image is obtained during afirst period of time, the second image is obtained during a secondperiod of time, the reference image is obtained during a third period oftime that is prior to the first period of time and the second period oftime; wherein the processor estimates a first relative change inposition of the scanning area of the subject by comparing the referenceimage and the first image; wherein the processor estimates a secondrelative change in position of the scanning area of the subject bycomparing the reference image and the second image; wherein theprocessor estimates a third relative change in position of the scanningarea of the subject by comparing the first image and the second image;wherein the processor calculates a quality metric of estimation of thesecond relative change in position based on: the first relative changein position; the second relative change in position; the third relativechange in position; and wherein the processor compares the qualitymetric to a threshold, in a first case wherein the quality metric isless than the threshold, adjusting position of the scanning area basedon the second relative change in position; and in a second case whereinthe quality metric is not less than the threshold, obtaining a newsecond image during a new second period of time.